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ABSTRACT 



Pursuant to the Council of Ontario Universities' procedures 
for periodic quality reviews of undergraduate programs, auditors examined the 
policies and programs of Brock University (BU) during 1997. Three departments 
were selected for audits of their program review procedures: Philosophy, 
Geography, and Physics; additionally, the Oenology and Viticulture Program 
was selected as a sample of a new program. After examining BU's internal 
review policies and the complete record of the program reviews, and finding 
evidence that mechanisms were in place to implement recommendations, the 
auditors held a site visit. Following a brief introduction and description of 
the methodology used in the audit, the report includes: (1) a description of 

BU program reviews; (2) the general principles and guidelines used for BU 
undergraduate reviews; (3) conformity of the BU review process for existing 
undergraduate programs with audit committee procedures; (4) conformity of BU 
policy for implementing new undergraduate programs with the audit committee 
framework for new programs; (5) the report’s conclusions, highlighting BU's 
close conformity with the audit committee framework; and (6) a summary of 
recommendations and suggestions. (Contains the schedule for the first cycle 
of audits (1997-2003) , lists of auditors and BU principal documents, and site 
visit schedule.) (MAB) 
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Glossary of Acronyms and Terms 

CAP Committee on Academic Policy, a Standing Committee of the Senate of 

Brock University 

COU Council of Ontario Universities 

MET Ministry of Education and Training 

OCAV Ontario Council of Academic Vice-Presidents 

OCGS Ontario Council on Graduate Studies 

UPRAC Undergraduate Program Review Audit Committee 



UPRAC Procedures Undergraduate Program Review Audit Committee Procedures refers to 

the OCAV Constitution, Bylaw 1 , Appendix A. 

UPR Framework . . . Undergraduate Program Review Framework refers to the Objective, 

Structure and Elements of the Undergraduate Program Review Procedure 
found in the OCAV Constitution, Bylaw 1, Appendix A, Section 1. 
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1 Introduction 

In October 1996, the Council of Ontario Universities (COU) approved the establishment of a 
procedure for the systematic auditing of the policies and processes in place at all Ontario universities 
for the conduct of periodic quality reviews of undergraduate programs. The procedure and guidelines 
specify that auditing of processes includes the examination of a representative sample of the quality 
reviews. Subsequently, in February 1997, the guidelines were amended to include the auditing of the 
mechanisms used by the universities for the implementation of new undergraduate programs. 
Authority for the organization and management of the audits is vested in the Ontario Council of 
Academic Vice-Presidents (OCAV). The detailed oversight of the audit procedure is devolved to a 
committee of OCAV, the Undergraduate Program Audit Review Committee (UPRAC), whose 
responsibilities are set out in Bylaw 1 of the OCAV Constitution. One of UPRAC’s duties is to 
recommend to OCAV the schedule of audits, which started in 1997 with Brock University and the 
University of Ottawa. The schedule for the first cycle was approved by OCAV at its meeting of 
February 27, 1997 and is displayed in Appendix 1. 

The audits themselves are conducted at arm’s length by at least three auditors who are appointed by 
UPRAC according to the criteria in the bylaw: “Auditors shall be chosen for their recognized strength 
in the development and operation of undergraduate programs. They shall not hold an administrative 
appointment in an Ontario university during their terms as auditors.” (See Appendix 2 for the names 
and affiliations of the Auditors for 1997.) The procedures to be followed by the Auditors are spelled 
out in the Procedures for the Audit of Undergraduate Program Reviews, Appendix A of the bylaw, 
hereafter called the UPRAC Procedures. It describes in some detail “the objective, structure and 
elements” that “any credible periodic undergraduate program review procedure undertaken by an 
institution must include.” For convenience, these key review components can be described as the 
Undergraduate Program Review Framework, and will be referred to hereafter as the UPR 
Framework. 

A critical caveat having to do with the timing of the introduction of the UPRAC Procedures and the 
first few institutional audits must be entered here. Those universities whose review policies and 
practices are being audited in the early years of the first cycle of audits will have had in place review 
policies that predate the approval of the UPRAC Procedures. This means that the program reviews 
sampled for the audit will have been conducted following institutional policies and processes that may 
not incorporate every aspect of the UPR Framework. Within these constraints, the Auditors could 
not apply the UPR Framework retroactively as a standard against which the university’s practices 
must be measured. The components of the UPR Framework have, therefore, been used as markers 
of “best practice” and as signposts for improvements. Specific suggestions and recommendations are 
accordingly made with a formative intent. 
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In framing the report and presenting their findings, the Auditors have found it helpful to distinguish 
between recommendations and suggestions. Instances where the Auditors consider the policies and 
procedures not to be in conformity with the UPR Framework are cast as recommendations. They 
have made suggestions in cases where, although the review process is in conformity with the UPR 
Framework, they think the process could be improved. It will be noted that some recommendations 
and suggestions are, in fact, generic in nature and may be relevant to many institutions. 

The Auditors also wish to record that they have found some parts of the UPR Framework to be open 
to interpretation, and for that reason they have applied it with “interpretive charity.” It should be 
emphasized that in no way do they think that this has weakened the audit process. To the contrary, 
just as there is room for improvement in the way in which institutions conduct their quality reviews, 
so too the UPR Framework may be improved. The Auditors will be reporting separately to OCAV 
on those aspects of the procedure that should be modified or clarified. 



2 Methodology and Verification Steps 

In preparation for their work, the Auditors requested from Brock University a copy of the current 
policies for conducting undergraduate program reviews and implementing new programs, plus the 
schedule of programs that had been reviewed and those slated for future review. 

To satisfy the injunction in the UPRAC Procedures that “the Auditors will take into account the 
diversity of educational programs offered by the university, whenever possible,” the reviews of three 
Brock programs were chosen for the audit, one from the humanities (Philosophy), one from the social 
sciences (Geography), and one from the physical sciences (Physics). The Oenology and Viticulture 
Program was selected as an example of a new program. 

As outlined in the UPRAC Procedures, the university was requested to provide “general documents 
that present the [undergraduate program] review practices and policies adopted and implemented by 
the institution” and “any additional documents they deem appropriate.” Relevant documentation 
would include: 

• the internal review policies in place at the time of the review of each program where they 
differed from those currently in force; 

• the complete record of the reviews chosen for audit: 

• the programs’ self-appraisal documents; 

• the evaluations of external experts; 

• the appraisals by peers internal to the university but outside the program under 
review; 

• the responses, if any, to the evaluations at all levels; 

• the minutes of any discussion at Senate (or one of its standing committees); and 
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• evidence that mechanisms are in place to act on the recommendations arising from the 
process. 

(A list of the principal documents provided by Brock University is included in Appendix 3.) 

Having met and reviewed the documentary record, the Auditors paid a site visit to the university on 
May 8, 1997. During the visit, they met with the Acting President, the deans of the three faculties, 
and the chairs, and in some cases faculty members, of the departments whose program reviews had 
been selected for audit. There was also a session with the chair and members of the Committee on 
Academic Policy and the Chair of the Senate. (The schedule of meetings for the site visit is provided 
in Appendix 4.) 

The Auditors would like to record their appreciation for the assistance and cooperation they received 
from all those involved with their visit and the provision of documentation. Everyone concerned 
approached the task with an understanding of its importance; it was viewed not as an intrusion, but 
rather as an opportunity to reflect on what it is that universities can and should be doing to improve 
their undergraduate programs. 



3 Program Reviews at Brock University 
3.1 Existing Programs 

For many years, undergraduate programs at Brock University have undergone quality reviews 
which included the use of external evaluators. Initially, the reviews tended to be conducted on 
an ad hoc basis at the instigation of a department chair. In 1 992, the Vice-President, Academic, 
introduced more systematic organization of reviews on a university-wide basis, with standardized 
terms of reference and guidelines for external reviewers that are set out in the document entitled 
Department or Program Review. These terms of reference and guidelines were those that were 
in effect at the university when the programs sampled for the audit were reviewed. 

In February 1996, the Brock Senate approved a more formal policy. External Review of 
Undergraduate Programs. This new policy predates slightly the adoption of the UPRAC 
procedures. Nevertheless, both the policy and the earlier practices and guidelines anticipated 
many of the review components set out in the UPR Framework. The Auditors had both the 1 992 
and 1996 procedures available and they will be discussed in sections 4 and 5. 

From the time of the introduction of the more systematic approach to reviews in 1992 to the 
approval of the 1996 Senate policy, 1 5 reviews were conducted. The university has established 
a schedule for the complete first cycle of program reviews by 2003 using the 1996 protocol; two 
have been completed, 30 are scheduled, and three new programs must be added (see the table 
that follows). 
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Schedule of Reviews of Existing Undergraduate Programs at Brock University 



Faculty 


Completed 
1992-19956 | 


Under February 1996 Policy 


Completed or 
underway 1996-97 £ 


Planned 

1997-2003 


Business 


— 


1 


— 


Education 


— 


— 


1 


Humanities 


4 


— 


11 


Mathematics and 
Science 


5 


— 


6 


Physical and Health 
Education 


— 


— 


3 


Social Sciences 


6 


1 


9 


Total 


15 


2 


30 



At Brock, interdisciplinary programs are reviewed either as freestanding programs or as part of 
the reviews of the academic units that have responsibility for the programs. 

Brock’s policy statement, External Review of Undergraduate Programs, envisages that “each 
Department should be reviewed at least once every seven years.” 

3.2 New Programs 

According to the guidelines set by the Ministry of Education and Training (MET), the 
implementation of new programs in Ontario universities should be preceded by their introduction 
in universities’ five-year plans. Once introduced in the five-year plan, new programs are 
implemented by universities as resources permit and opportunities arise. Therefore, the time 
frame during which new programs pass from the stage of introduction in the university’s five- 
year plan to implementation varies. Three new programs were introduced recently by Brock 
University. 
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4 General Principles and Guidelines Used for Undergraduate Reviews at Brock 
University 

4.1 Existing Programs 

Brock’s policy governing the review of its undergraduate programs is contained in the document 
entitled External Review of Undergraduate Programs, which was revised by Senate in February 
1996. It sets out the process to be followed and is the authoritative basis for conduct of reviews. 
A second document (undated) entitled Department or Program Review, which was introduced 
in 1992, continues to be available for use as a source of guidance for external evaluators. It 
contains a list of questions organized under a series of headings on which external reviewers 
might be expected to comment. 

The External Review of Undergraduate Programs policy statement begins by enunciating the 
principle that each department will be reviewed at least once every seven years, with the 
sequence determined by the dean in consultation with chairs and directors. Such reviews examine 
the programs offered by the department and, where relevant, the links to other departments that 
offer support to these programs, or with which the department has interactions. 

One year in advance of a review, the dean is charged to choose reviewers “from other Canadian 
universities, although representatives of industry and business may also be suitable in some 
cases” again in consultation with the department under review. (The number of reviewers may 
vary depending on the discipline.) 

In advance of the site visit, the department is responsible for preparing materials to “include the 
undergraduate calendar, outlines and other handouts for all courses, a record of enrollment over 
three years in all courses, information on teaching loads, curricula vitarum [sic] of all faculty, 
graduation rates and any other information which is appropriate.” 

The dean is responsible for “specific” terms of reference, but the responsibilities of the reviewers 
generally include scrutiny of the departmental materials, site visit and preparation of a report. 
The Brock procedures make a special point of additionally charging external reviewers with 
responding to queries arising from their reports. 

The policy suggests an itinerary for the site visit, which should last “at least one and a half days” 
and include meetings with faculty, staff, students, teaching assistants and the chair of the 
department. Representatives from the library and from other units with which the department 
has joint programs or other interactions should also be included. The dean is expected to meet 
with the external reviewers at the beginning and at the end of the visit. The Vice-President, 
Academic, and “heads of administrative units providing significant support to the academic unit 
under review” are also interviewed. 
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The policy goes on to illustrate the kinds of issues reviewers should address in the written report. 
The report should include critical comments on: 

• the curriculum: 

• its stated goals; 

• its modernity; 

• the skills it provides to students; 

• its gender equity; and 

• its flexibility to accommodate differing student needs 

• the significance of enrolment patterns within the department 

In addition to these issues, deans may also draw from the questions contained in the document 
Department or Program Review referred to above. 

Reviewers tender their reports to the dean, who in turn transmits the reviews to the department 
and “to other departments that participated in the review such as those involved in joint 
programs.” Departments are enjoined to discuss the reviewers’ recommendations and “submit 
questions or requests for clarification to a reviewer.” By way of follow-up, “the Dean will meet 
with the Chair and/or the department to consider implementation of recommendations which may 
be appropriate.” 

The dean then transmits the reviewers’ reports to the Senate for referral to the Committee on 
Academic Policy (CAP), which discusses the reports and the departmental response, and may 
request that departments and/or the faculty dean “report back to CAP about progress on 
implementation of appropriate recommendations in the report.” 

Finally, CAP annually provides a summary and copies of the external reviews to the Senate, 
which discusses the summary report “and, when appropriate, transmit(s) the document to the 
university President for distribution to the Board of Trustees.” 

4.2 New Programs 

The university’s policy and practice for the introduction of new programs is now guided to a 
large extent by the current requirements of MET as set out initially in the memorandum dated 
November 8, 1996 from the Assistant Deputy Minister to the Executive Heads and reconfirmed 
in a subsequent memorandum on June 24, 1997 from the Director of the Universities Affairs 
Branch . The programs approval process is described as one of self-administered regulation and 
calls for the institutions to certify that their proposals meet six criteria: program nomenclature 
(or “truth-in-advertising”), academic quality, financial viability, institutional appropriateness, 
student demand and societal need. A seventh criterion, duplication, is evaluated by the Ministry. 
(“Core” undergraduate Arts and Science programs are exempted from this process.) 
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Internally at Brock, CAP is charged in its terms of reference [Faculty Handbook Section II: 9.4. 1 
a) and b)] with the responsibility for making recommendations to the Senate on “the academic 
philosophy of Brock University in relation to the consideration of new programs, criteria for the 
evaluation of such programs, [and] the establishment of program priorities and relationships 
between existing or proposed programs of study.” The Standing Committee also recommends 
for approval by the Senate all new undergraduate degrees and programs. Before being presented 
to CAP, new programs are normally reviewed by the appropriate departments and faculty 
undergraduate curriculum committees. 



5 Conformity of the Brock University Review Process for Existing 
Undergraduate Programs to the Expectations of the UPRAC Procedures 

The UPRAC Procedures specify that the report of the Auditors will include comments on how well 
the procedures and process of the institution under review conform to the objectives, structure and 
elements they establish (the UPR Framework). They also specify that the Auditors should provide 
details on “how well the review procedure and process conform to the institution’s own policies.” 
This provision is especially important at the inception of the audit process. 

As all the program reviews selected for audit in 1997 were conducted prior to the adoption of the 
UPRAC Procedures, a negative judgment relative to that standard would be both uninformative and 
unfair, since the local practices could not have been designed with the UPR Framework in mind. 
Second, it was evident to the Auditors that local review procedures are in a state of flux, and that 
actual practice has in many regards outstripped the policy statement. Finally, while the expectation 
is that all Ontario institutions will generate program review policies that reflect at least the minimum 
requirements of the UPR Framework (if they have not already done so), it is now, and will doubtless 
always be, the case that, to accommodate universities’ diverse needs, local review policies go beyond 
the minimum that the UPR Framework requires in different ways that call for verification. 

These comments on conformity are organized into two parts: 

• conformity of Brock University’s procedures to its policy in the period 1992-1996; and 

• conformity of Brock University’s policies for program reviews to the UPR Framework. 

In the first part, the comments deal with the practice that was followed in the period 1992 to 1995 
during which time the reviews selected for audit at Brock University took place. The second part 
assesses the February 1996 policy, which was developed before the UPRAC Procedures were 
finalized. 
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5.1 Conformity of Brock University’s Procedures for Program Reviews to its PoliiaLin 
the Period 1992 to 1995 

To anticipate the conclusion reached below, Brock’s procedures and process — in all matters 
that the Auditors have been able to verify — from 1 992 to 1 995 appear to have anticipated the 
policy it adopted formally in 1996, which in turn incorporates to a large extent the components 
of the UPR Framework. The Auditors were impressed by the obvious commitment of those 
they met at Brock to the review process and to making it effective. 

The deans confirmed that they assumed responsibility for sequencing the reviews and for 
choosing the external reviewers. They explained the considerations that bore on their decisions. 
They also confirmed that, in addition to the materials specified in the policy, they ask 
departments to prepare a brief for the external reviewers; this was already evident from the 
documentation provided in response to our request. Another feature on which the deans insisted 
was the formative character of the review process. Finally, in their own reflection on what 
worked and what did not in the review process, they were in general agreement that much of the 
data on which reviews depended were, at a minimum, difficult to access and difficult to use. 

Suggestion 1: The difficulty in making relevant data and analysis 
available to those involved in the review is by no means unique to 
Brock. If, however, reviews are to be meaningful, and if 
recommendations and decisions on whether or not to accept them are 
to be based on evidence, it is vitally important that Brock and other 
universities address the perceived difficulties in the manner that best 
accords with their individual needs and organization. 

The Auditors found that, in virtually every example that they were able to probe, 
recommendations made by external reviewers were taken seriously, even when they were not 
ultimately accepted. The Auditors state that, in their view, “serious consideration” is the 
appropriate standard to bring to bear in assessing the responses of units under review to 
formative recommendations, which are counsel and not command. In one instance where a 
query was referred back to reviewers, the department rejected a recommendation about its first- 
year teaching formats that would have saved money because it was incompatible with the 
department’s informing philosophy and would, in its judgment, have compromised quality. 

Two out of three audited reviews were scrupulously organized around the 10 categories in the 
Department or Program Review (1992) guidelines. (In one case, the same extra dimension 
highlighting a particular local issue was added by the two reviewers). In the third case, the dean 
charged the single external reviewer with responding to three questions, which are related, but 
not identical to, the questions in the Department or Program Review document: viz., “Is the 
undergraduate sequence of courses suitable for majors? Is the course content modem and 
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appropriate? Are the “service courses” provided by the Department meeting the needs of 
students from other Departments?” 

The Auditors conclude, based on examination of the documentation provided and interviews 
with interested parties, that the procedures and process in place at the time the reviews were 
conducted were generally observed in the three cases that were audited. 

5.2 Conformity of Brock University’s Policy for Program Reviews to the UPR 
Framework 

In making recommendations and suggestions for improving what are already a sound policy and 
procedure, the Auditors once more wish to emphasize their awareness that the 1996 policy — 
and the program reviews that were audited — predate the UPRAC Procedures. Though none 
of the audited program reviews were conducted under the 1996 Brock policy, this policy 
incorporates much of the practice that was followed between 1992 and 1995 and much that is 
in the UPR Framework. The conformity assessment evaluates the policy against the objective, 
structure and elements of the Undergraduate Program Review Procedure contained in the 
UPRAC Procedures. These components of the UPR Framework appear in italics below. 

5.2.1 Objective of the Undergraduate Program Review Procedure 

The objective of the institutional periodic undergraduate program review procedure is 
to assess the quality of the undergraduate programs that the university provides in all 
areas of study. 

The Brock policy does not explicitly state the objective for the review procedure though it was 
obvious at the meetings with the Auditors that all those involved clearly understood the essential 
purpose of the review to be a means of ensuring a regular and formal assessment of quality. 
Most often, the appropriate and convenient academic entity for review is the department or a 
similar organizational unit. Yet it is the quality assurance of the undergraduate program review 
process that is to be audited. Programs may be, and often are, co-terminus with departments, 
and institutions may, for a number of reasons, wish to incorporate program reviews into the 
reviews of the organizational units that offer the programs. This point may be obvious, but it is 
made here in order to emphasize that institutional program review processes (including the 
policies and procedures) should deal explicitly with those programs that span departmental lines. 
Interdisciplinary programs are the most common of these, and they are a major feature of the 
Brock academic landscape. 

Though the policy specifies that external reviewers should meet “other Departments that have 
joint programs or other interactions with the Departments under review” and that copies of the 
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reviewers’ reports should go to those “other Departments,” it could, nevertheless, be more 
explicit in its reference to interdisciplinary programs. 

It is also important to ensure that any graduate program offered by the department/academic unit 
whose undergraduate program is being evaluated does not overshadow the review of the 
undergraduate program. 

Recommendation 1: The Brock policy should be amended to include 
a clear statement of the objective(s) for the reviews of existing 
undergraduate programs. Given the importance of interdisciplinary 
programs to the university, the policy should indicate clearly how they 
are reviewed. 

5.2.2 Structure of the Undergraduate Program Review Procedure 

The institutional undergraduate program review policy statement should: 

(1) identify an authority responsible for application of the policy; 

The deans have a major role in the “application of the policy” and, independently, they are 
responsible for the reviews within their faculties. However, it is not clear from the External 
Review of Undergraduate Programs document that they are the “authority” in the meaning of 
the UPR Framework. The Vice-President, Academic, took a lead in the move to systematize the 
review process and is kept informed by the deans of reviews that are planned and underway, but 
the Vice-President, Academic, does not have a specified role in the review process — other than 
meeting the external reviewers). In some instances it would appear that the actions of the deans 
are subject to the oversight of CAP, which may request a report from the appropriate dean 
“about progress on implementation of appropriate recommendations in the report.” Ultimately, 
a summary by CAP and copies of the external reviews are discussed by the Senate, but only on 
an annual basis. 

Suggestion 2: The absence in the policy of reference to a central 
authority with ultimate responsibility for the reviews need never cause 
a problem at Brock, since it is clear to all of the parties concerned that 
the review process is vital to the academic health of the institution, and 
the Auditors are satisfied as to their commitment to the process. 

However, since the Auditors understand that CAP is disposed to 
review the policy document in fuller light of the UPR Framework, they 
commend this matter to their attention and that of the Vice-President, 

Academic. 
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(2) define the undergraduate program review criteria; 

The UPR Framework requires that institutions be explicit about the issues to be addressed in the 
review process. In this sense, Brock’s Department or Program Review document is most helpful 
and continues to guide the deans as they draft the terms of reference for external reviewers. The 
current Brock policy document also lists some important questions that should be addressed. 
(The subjects raised in these two documents might equally be put to the departments under 
review — see section 5.2.2 (3) a) below.) 

Suggestion 3: The document Department or Program Review or some 
version of it might usefully be added as an appendix to the policy 
document in order to more explicitly “define the undergraduate 
program review criteria.” 

(3) prescribe at least the following steps: 

a) self -appraisal by professors and students participating in the program; 

As indicated above, though a self-appraisal is not required by Brock’s policy, in practice deans 
have encouraged departments to conduct a self-evaluation and to provide the results to the 
external reviewers. It remains the case, however, that the lack of both a policy requirement and 
some guidance to departments under review about what is required and how best to achieve it 
are the most serious deficiencies in the Brock protocol and call for urgent corrective action. 
While the serious commitment of all concerned at Brock to the review process cannot be 
doubted, it is clear to the Auditors that the effectiveness of some of the actual reviews and the 
realization of their full potential as formative exercises were severely hampered by the lack of 
policy or guidelines or both. 

The importance of self-appraisals and the need for guidelines on their purpose and content was 
illustrated to the Auditors by two of the reviews that were audited. In one, a serious miscue 
between department and external reviewers might have been avoided by having the unit frame 
the agenda with a self-reflection. In another, a unit bent over backwards to provide “some basic 
and (hopefully) non-controversial facts,”deliberately eschewing “assessment from the 
department’s point of view, of the program’s successes and failures,” which allowed it to avoid 
confronting serious issues that were raised by the external reviewers. 

Finally, the UPR Framework looks for self-appraisal by professors and students. Students did 
seem to be involved in program reviews at Brock by meeting with external reviewers, but not 
otherwise. 
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Recommendation 2: The Brock policy should be amended to require 
self-appraisal by professors and students participating in the program. 

One possible approach might be for departments and programs under review to produce a report 
on the same issues that external reviewers are expected to address, with specific questions or 
suggestions for reviewers’ feedback. In the Auditors’ view, the best time for the self-appraisal 
is before the instructions are given to the external reviewer(s). Additionally, the Senate CAP, 
or the Brock administration, should consider sponsoring workshops to assist units in the task, 
at least while the requirement is new and unfamiliar. Guidelines might be written and an 
individual or committee could be designated as a source of advice. If units offer interdisciplinary 
programs, the other units involved in formulating and delivering the program should participate 
in the self-appraisal. 

b) evaluation by at least one expert from another university, including universities 
outside Ontario; 

This requirement is part of the Brock policy statements, and it was the practice in all the reviews 
that were audited. While the campus visit is well described, again, guidelines for the external 
reviewers would be useful. 

Recommendation 3: Though there is no evidence to question the 
impartiality of the external evaluators who have been chosen by Brock 
University, the auditors recommend that all Ontario universities that 
have not already done so enunciate as part of their program review 
policy the principle that external evaluators will be at arm’s length from 
the program under review, along with their interpretation of what is 
entailed by the principle (not relatives, collaborators, 
supervisors/supervisees, and so on). 

c) appraisal by peers chosen from among professors from the institution who do not 
participate in the program under review ; 

This requirement in the UPR Framework calls for an interdisciplinary perspective to ensure that 
university-wide standards for review are met in all cases. How this requirement is met by 
universities may vary according to size, structure and ethos. At Brock, the Senate Committee 
on Academic Policy (CAP) plays this role, but it is the opinion of the Auditors that CAP’S role 
needs to be more clearly defined if it truly is to have an evaluative function. To illustrate, by the 
time the dean has sent the written report to the Senate (a procedural formality for submitting it 
to CAP), the dean will have already met with the academic unit “to consider implementation of 
Recommendations which may be appropriate” so the evaluative phase is over. It is the Auditors’ 
view that if CAP, or any other committee, is to meet the requirement for appraisal by peers, it 
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should be involved in the process at an earlier stage than at present. It would certainly be most 
appropriate for the dean to be part of this stage. 

Recommendation 4: The terms of reference of the Senate Standing 
Committee on Academic Policy should specify its function when it is 
considering the documentation from the review process. The 
documentation should include the departmental self-appraisal and the 
external evaluators’ reports. 

d) the university may also, where appropriate, seek the advice of others such as 
representatives of industry, professions and practical training programs ; 

Brock’s policy clearly allows for this. 

Suggestion 4: The policy should make it clear that there must always 
be at least one evaluator who is an academic even when the advice of 
others is sought. 

(4) specify the procedures for action on the recommendations arising from the 
undergraduate program review; 

There is provision for discussion within the department, discussion with the dean, and the 
possibility of a progress report requested by CAP. What is lacking is a mechanism to summarize 
the action to be taken, which would be subject to monitoring and future assessment. 

Suggestion 5: The policy would be strengthened if it were to specify 
the steps that should follow the adoption of the recommendations 
arising from a program review. There should also be a requirement to 
report to CAP after a year or two. 

(5) determine the periodicity of the review cycle; this should not exceed 10 years for 
a university ’s full set of programs; 

The Brock policy complies, requiring programs to go through a review at least every seven 
years, and the review schedule meets that requirement. Since the review cycle just started, it is 
premature to assess whether the proposed cycle is being followed. 

(6) establish a policy to ensure publication of a summary of the institutional review 
report (excluding all personal information); the summary should include a statement 
of the strengths and weaknesses of the program, and an outline of the main 
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recommendations of the review committee. This summary is to be presented to the 
Senate and Board of Governors of the university. 

Brock has such a policy; however, it does not categorically require the summary report to go to 
the Board, but only “when appropriate, transmit the document to the University President for 
distribution to the Board of Trustees.” 

Recommendation 5: The policy should be amended to make the 
transmittal of the Senate’s summary of external reviews to the Board 
of Trustees standard practice in all cases. 

5.2.3 Elements of the Undergraduate Program Review 

The institutional procedure for review of existing undergraduate programs should 
address the following points: 

(1) consistency of the program with the general objectives of the institution ’s mission 
and development plan; 

The basic academic mission of the university was so well understood by everyone with whom 
the Auditors met that it may go without saying. As observed above, however, Brock’s mission 
may not be so well understood by the external evaluators, whose participation is an essential 
feature of the review process, and who need that understanding to discharge their function 
effectively. This element of a review is a valid criterion and one that needs to be explicitly 
included. 



Recommendation 6: An evaluation of the consistency of programs 
with the appropriate sections of the university’s mission statement 
should be added as a requirement of the review policy. 

(2) appropriateness of the admission requirements for the program ’s educational 

objectives; 

This is not a requirement of the February 1996 review policy. (It is noted that it was not a factor 
in any of the three programs whose reviews were selected for audit, with the possible exception 
of Physics whose students would require a certain level of preparation in mathematics in order 
to have a reasonable chance of success.) 

Recommendation 7: The university review policy should include a 
reference to the evaluation of the appropriateness of admission 
requirements where academically warranted. 
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(3) appropriateness of the program ’s structure and curriculum for its educational 

objectives; 

(4) appropriateness of mode of delivery and of methods used for the evaluation of 
student progress ; 

These requirements are covered in brief in the policy, though they are, with the exception of 
“methods used for the evaluation of student progress,” addressed at greater length in the 
document Department or Program Review which, as noted, still serves as a source of guidance 
for the drafting of the external reviewers’ terms of reference. Implementation of 
Recommendation 1 would give additional emphasis to them. 

(5) appropriate utilization of the existing human/physical/ financial resources; 

This element is covered both in the Workload, and the Facilities and Support sections of the 
Department or Program Review document, and as with several other “Elements,” should be 
added to the review criteria. 

(6) definition of indicators that provide evidence of: 

• quality of faculty ; 

• student clientele (applications and registrations); 

• student quality; and 

• outcomes of the program (graduation rate, length of studies, etc.). 

Among the details to be provided to the external reviewer are enrolment records, teaching loads, 
curricula vitae and graduation rates as well as “any other information which is appropriate,” but 
there is no specification of “indicators” as such — difficult as it may be to find common 
agreement on what they should be. Information is available and there is genuine willingness on 
the part of those who can provide it to do so and on the part of those who would use it to make 
better use of such data and analytical tools to interpret it. Brock conducts a post-graduation 
survey that may be a valuable source of information to feed into the review process. 

Suggestion 6: The review policy should provide more explicit 
description of the quality indicators that could form part of the review 
dossier. (See also Suggestion 1.) 

The issue of quality indicators is a matter of continuing concern. Criteria that measure the 
quality of incoming or departing students or some assessment of the “improvement factor” are 
issues that deserve consideration by OCAV. 
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6 Conformity of the Brock University Policy and Procedures for the 
Implementation of New Undergraduate Programs to the UPR Framework for 
New Programs 

The UPR Framework for the implementation of new undergraduate programs states that: 

(1) the institution should provide to the auditors its policy regarding the implementation of 
new undergraduate programs; 

(2) the institutional procedures for the implementation of new undergraduate programs 
should address points [a) - e)] above and provide evidence of the quality of the faculty. 

Much of what was said in the opening paragraph of section 5 holds here too. The Terms of 
Reference, found in the Brock Faculty Handbook, clearly set out the responsibilities of the Senate 
Standing Committee on Academic Policy for the oversight and approval of the introduction of new 
undergraduate programs. These Terms of Reference and the record of the approval of the new 
Oenology and Viticulture Program — the one selected for the audit — predate the establishment of 
the UPR Framework. The Auditors were presented with considerable documentation for the 
introduction of this program. They have no reason to question the thoroughness with which it was 
scrutinized, and the process appeared to follow the procedures Brock established for the introduction 
of new programs. However, more detailed guidelines are needed in order to allow for an audit of 
compliance. 

One way of viewing what OCAV has demanded in the UPR Framework is that, consistent with the 
information requirements now imposed by the Ministry of Education and Training for funding 
approval of new “non-core” programs, universities should be taking into account the consistency of 
what is proposed with their more general development plans. Specifically, they should address the 
issue of whether this is an appropriate use of available resources, as well as provide some critical 
scrutiny of the quality of the new activity (as gauged by curriculum, delivery, quality of faculty, and 
so on) following the requirements set out in the UPR Framework. 

Recommendation 8: The provisions governing the implementation of new 
programs at Brock University, as at all Ontario universities, should be revised 
consistently with both the Ministry of Education and Training information 
requirements that need not be supplied in detail but must be certified by the 
president of the university in funding applications as set out in the MET 
memorandum regarding Program Approvals Process on November 8, 1996, and 
the UPR Framework found in Appendix A 1.4. 
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7 Conclusion 

The Auditors wish to note that they believe the Undergraduate Program Review Audit Procedure 
reinforces and supports the commitment of Ontario’s universities to offering programs of the highest 
quality. It builds on the internal quality assurance processes that the institutions have put in place. 
Based on their own experience as senior academic administrators, the Auditors recognize the 
importance of such processes in maintaining and improving the quality of education that universities 
deliver to undergraduates. Also, they appreciate the difficulty the institutions face in times of fiscal 
constraint. The pressures are immense. 

It is to the credit of the institutions that they are committed to self-examination and renewal in order 
to offer programs of excellent quality. A regular, institution-wide series of undergraduate program 
reviews is a serious and demanding undertaking. These reviews are not the only ones that 
departments and other academic units undergo, and the demands on the faculty and staff are 
considerable. For this reason, the Auditors have been most impressed by how thoroughly and 
carefully Brock University and the University of Ottawa, the two institutions that were audited in 
1997, have approached their undergraduate review process. 

As a general conclusion, the Auditors are satisfied that, in regard to both policy and practice, Brock 
University’s process for undergraduate program reviews conforms closely to the UPR Framework. 
As already observed, the university’s practice goes beyond what is required by its policy. They have 
noted with recommendations and suggestions where the policy might be improved and made to 
conform more closely with the UPR Framework. The most significant of these are the 
recommendations having to do with the requirement for a program’s self-appraisal, the role of the 
Senate committee, CAP, and follow-up to the reviews. 

8 Summary of Recommendations and Suggestions 

Finally, for ease of reference the Recommendations and Suggestions are repeated here: 

Recommendations: 

Recommendation 1: The Brock policy should be amended to include a clear statement of the 
objective(s) for the reviews of existing undergraduate programs. Given the importance of 
interdisciplinary programs to the university, the policy should indicate clearly how they are 
reviewed. 

Recommendation 2: The Brock policy should be amended to require self-appraisal by 
professors and students participating in the program. 
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Recommendation 3: Though there is no evidence to question the impartiality of the external 
evaluators who have been chosen by Brock University, the auditors recommend that all Ontario 
universities that have not already done so enunciate as part of their program review policy the 
principle that external evaluators will be at arm’s length from the program under review, along 
with their interpretation of what is entailed by the principle (not relatives, collaborators, 
supervisors/supervisees, and so on). 

Recommendation 4: The terms of reference of the Senate Standing Committee on Academic 
Policy should specify its function when it is considering the documentation from the review 
process. The documentation should include the departmental self-appraisal and the external 
evaluators’ reports. 

Recommendation 5: The policy should be amended to make the transmittal of the Senate’s 
summary of external reviews to the Board of Trustees standard practice in all cases. 

Recommendation 6: An evaluation of the consistency of programs with the appropriate sections 
of the university’s mission statement should be added as a requirement of the review policy. 

Recommendation 7 : The university review policy should include a reference to the evaluation 
of the appropriateness of admission requirements where academically warranted. 

Recommendation 8: The provisions governing the implementation of new programs at Brock 
University, as at all Ontario universities, should be revised consistently with both the Ministry 
of Education and Training information requirements that need not be supplied in detail but must 
be certified by the president of the university in funding applications as set out in the MET 
memorandum regarding Program Approvals Process on November 8, 1996, and the UPR 
Framework found in Appendix A 1.4. 



Suggestions: 

Suggestion 1: The difficulty in making relevant data and analysis available to those involved in 
the review is by no means unique to Brock. If, however, reviews are to be meaningful, and if 
recommendations and decisions on whether or not to accept them are to be based on evidence, 
it is vitally important that Brock and other universities address the perceived difficulties in the 
manner that best accords with their individual needs and organization. 

Suggestion 2: The absence in the policy of reference to a central authority with ultimate 
responsibility for the reviews need never cause a problem at Brock, since it is clear to all of the 
parties concerned that the review process is vital to the academic health of the institution, and 
the Auditors are satisfied as to their commitment to the process. However, since the Auditors 
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understand that CAP is disposed to review the policy document in fuller light of the UPR 
Framework, they commend this matter to their attention and that of the Vice-President, 
Academic. 

Suggestion 3: The document Department or Program Review or some version of it might 
usefully be added as an appendix to the policy document in order to more explicitly “define the 
undergraduate program review criteria.” 

Suggestion 4: The policy should make it clear that there must always be at least one evaluator 
who is an academic even when the advice of others is sought. 

Suggestion 5: The policy would be strengthened if it were to specify the steps that should follow 
the adoption of the recommendations arising from a program review. There should also be a 
requirement to report to CAP after a year or two. 

Suggestion 6: The review policy should provide more explicit description of the quality 
indicators that could form part of the review dossier. (See also Suggestion 1.) 
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Appendix 1 — Schedule for Program of Audits in First Cycle: 1997-2003 



Year iii 
Cvcle 


Calendar | 
^Year 




Universities 




1 


1997 


Brock 


Ottawa 


— 


2 


1998 


Western 


Windsor 


— 


3 


1999 


Carleton 


Queen’s 


Nipissing 


4 


2000 


Lakehead 


York 


— 


5 


2001 


Ryerson 


Toronto 


Trent 


6 


2002 


Laurentian 


Waterloo 


— 


7 


2003 


Guelph 


McMaster 


Wilfrid Laurier 



o 

ERIC 
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Appendix 2 — List of the Names of the Auditors 
Dr. Naomi Griffiths: 

Adjunct Research Professor, Department of History, Carleton University; Dean, Faculty of 
Arts, Carleton University, 1979-87 

Dr. Ross Rudolph: 

Professor, Department of Political Science, York University; Associate Dean, Faculty of Arts, 
1985-88; Associate Vice-President (Faculties), York University, 1990-1996 

Dr. Edward Stansbury: 

Professor Emeritus, Meteorology, McGill University; Dean, Faculty of Arts & Science, 1969- 
71; Dean, Faculty of Science, 1971-76; Vice-Principal, Planning, McGill University, 1976-86; 
Member, la Commission de verification de F evaluation des programmes, Quebec 
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Appendix 3 — List of the Principal Documents Provided by Brock University 

External Review of Undergraduate Programs. Policy document approved by the Senate. 
February 1996. 

Department or Program Review. Guidelines for external reviewers. 1992. 

Faculty of Physical Education and Recreation, Revised Procedures on Undergraduate Program 
Review. Undated. 

Faculty Handbook, Sections II 9.4, “Standing Committee on Academic Policy” and Section III 7.9, 
“Program Approval.” 

Cade, William H. Letter from the Acting Vice-President, Academic, containing the schedule of 
reviews of undergraduate programs. April 14, 1997. 

Documentation for the reviews of the undergraduate programs in Geography (1994), Philosophy 
(1994-95) and Physics (1994-95) includes: letters to the external reviewers from the deans, 
reports from the external reviewers, department profile (1), letters (2) to the deans containing 
comments of the chairs, and extract from the minutes of the meeting of Senate, May 14, 1997. 

Documentation for the implementation of the new undergraduate program in Oenology and 

Viticulture includes: “Oenology and Viticulture Proposal”, the Report to Senate of the 
Committee on Academic Policy, April 30, 1996, and extract from the minutes of the meeting of 
Senate, May 8, 1996. 
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Appendix 4 - 


— Schedule for the Site Visit 


9:00 a.m. 


Dr. Susan Clark, Acting President 


10:00 a.m. 


Dean Will Webster, Faculty of Social Sciences 

Dean John Sivell, Faculty of Humanities 

Acting Dean Rick Cheel, Faculty of Mathematics and Science 


1 1:00 a.m. 


Professor John Mayer, Chair, Department of Philosophy 


1:30 p.m. 


Professor Keith Tinkler, Chair, Department of Geography 


2:30 p.m. 


Professor Fereidoon Razavi, Chair, Department of Physics, and representatives from 
the department 


3:30 p.m. 


Professor Jack Miller, Chair, Senate Committee on Academic Policy 

Members of the Committee: Professors Barry Joe, Garth Stevenson, Angus Somerville 

and Merle Richards 

Professor Ken McKay, Chair of Senate 
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